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(57) Abstract 

The present invention refers to a 
method and a system in a virtual studio for 
optical non-contact positioning of at least 
one camera (11, 31) in a volume by means 
of one or more tracking cameras (18, 38). 
The camera (11, 31) is equipped with a 
marker unit (19, 39), which is visible at least 
from one tracking camera (18, 38), which 
generates a signal containing information 
about the position and/or orientation of 
the camera (11, 31). Said information is 
transferred to a unit for further processing 
with respect to a second set of information 
for generating an essentially coherent main 
information. 
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TITLE 

SYSTEM RELATING TO POSITIONING IN A VIRTUAL STUDIO 

TECHNICAL FIELD 

The present invention refers to a system and a method in a so-called virtual studio for 
determination of the position of at least one camera in a volume. 

BACKGROUND OF THE INVENTION 

A new technic largely used within the television is so-called "virtual studios", which means that 
the entire or parts of a scene and properties are generated through computer graphic. The scene 
created in the computer is then overlayed with "real" images (video), for example, the picture of 
an actor, so that in the final picture, the actor appears to be in the computer generated scene. 

That is extremely important that the real video and the computer graphics fit. The data system that 
is used to assemble the images must have information about the position and orientation of the 
video camera, i.e., the position of the real and virtual camera must be identical. 

The second problem that occurs is that the present tracking systems are based on cameras 
equipped with CCD in combination with image processing units. Usually, the information is read 
from the CCD-chip serially by means of one to four serial shift registers. The maximal reading 
speed is limited, e.g. to 20 MHz. By means of two shift registers and one CCD with 500x600 
pixels, the readout takes 15 ms for one image. A faster readout normally degenerates the signal-to- 
noise ratio (SNR) of the (analogous) output. Consequently, the number of pixels for a given 
maximal computing period is limited, and accordingly the best possible accuracy. It becomes 
difficult or impossible to discover small movements of a measured body. 

DESCRIPTION OF PRIOR ART 



The technique for positioning a camera and its orientation is called "motion tracking" or camera 
tracking". Mechanical methods have been used in the past, for example angle indicators on a 
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stationary support, to receive information about the orientation of the camera. 

Lately, non-contact methods have been developed. One system uses "bar-codes" arranged on the 
studio walls for position and angel determination. The bar-code comprises a pattern of two 
5 different shades and enables extraction in real time of the camera parameters from the film (the 
video signal), which is then processed in a video processor. — 

Another system uses an extra camera attached on the TV camera and directed towards the roof of 
the studio where a coded disc is provided, which can be detected by the camera and after the 
10 image processing, the position of the TV camera is indicated. 

WO 97/1 1 186 discloses a positioning and orientation system for an object, in which the markers 
for example formed as frames are attached on a surface such as a roof and the position and 
orientation of a camera is determined, e.g. with respect to the markers. 

15 

The position of an object is known, e.g. through US 5,227,985, in which the position of an object 
provided with a marker is determined by means of a camera. 

BRIEF DESCRIPTION OF INVENTION 

20 

The object of the present invention is to provide a system for a non-contact positioning of at least 
one camera in a volume in real time and in a very simple and accurate way. 

The system according to the invention enables free relocation of a TV camera above a surface 

2 5 without influencing the generation of position and orientation parameters. 

The system according to the invention allows both high positioning speed and accuracy. 

Additionally, the system according to invention provides following advantages: 

3 0 - measurement and positioning in large volumes, 

good precision, i.e. a small change in the position or angel can be detected, e.g. 0.1 mm for 
lateral offset and 0.01 degrees for angular, 

low noise (fluctuation) in the obtained position information from of the system, 
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low operation (i.e. slow displacement of the measured position without the measured 
object being displacement), 
- high measuring speed, i.e. many measurements per second, e.g. 60/s, 

quick response, i.e. each measurement result is available within a certain period, good 
detection performance, i.e. detection of small markers within large distance, and enough 
redundancy and consequently high reliability during all the operation times. - 

These tasks have been solved by means of the initially mentioned camera being equipped with a 
marker unit, which is visible at least from a tracking camera, which generates a signal containing 
information about the position and/or orientation of the camera, which information is supplied to a 
unit for further processing with respect to a second set of information for generating an essentially 
coherent main information. 

In one embodiment, the tracking camera is a CCD-equipped camera and operates within IR- 
region. Preferably, the tracking camera communicates with a computer unit, which also is 
connected to the camera for positioning the camera. 

The tracking cameras are arranged to measure the marker unit whereby the information about the 
degrees of freedom of the marker units is determined. The camera's position is determined 
relative to a global coordinate system of the volume. 

According to a method of the invention for a non-contact positioning of at least one camera in a 
volume by means of one or more tracking cameras, the method includes the steps of providing the 
camera with a marker unit visible at least from some of the tracking cameras, and generating a 
signal containing information about the degrees of freedom of the camera by means of the 
tracking camera. 

The method further includes measuring in an image recording unit of the camera and relating the 
generated degrees of freedom to said measurement. 

Advantageously, the measurement of the tracking cameras can be conducted according to an 
automatic procedure whereby a special marker unit is attached onto a lens mount of the TV 
camera, after which the tracking camera system measures with respect to the marker unit relative 
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the marker provided on the camera during the entire measurement procedure. 

In an embodiment, said camera is provided with at least one motion sensor in form of an inertial 
detector and/or accelerometer and/or gyroscope for an inertial-based positioning. However, 
5 several sets of motion sensors can be arranged in different positions on a body to be measured to 
compute angel data (role, par and tilt). The optical positioning provides an absolute measurement 
while the inertial-based positioning provides a relative measurement. Upon interference in one of 
the optical or inertial-based positionings, data from the other positioning is used. 

1 0 BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is described with reference to attached drawings showing an embodiment, in which: 
Fig. 1 is a very schematic view of the system according to the invention, 
Fig. 2 is a schematic marker unit, which can be used in a system according to the invention, 
15 and 

Fig. 3 is a second embodiment of a system according to the invention. 

DESCRIPTION OF THE EMBODIMENTS 

2 0 According to the invention a camera system is used to measure the position and orientation of one 
or several TV/video cameras in a studio (or any place for filming). Each TV camera to be 
traced/positioned is provided with at least one marker and is measured in, in real time, by means 
of an arrangement specific for the system. The information about the position and orientation of 
the TV camera is then transferred, in real time, to a data system that controls the virtual camera, 

2 5 i.e., a camera which in the virtual studio corresponds to the real camera. The term "TV camera" is 

used for simplicity reason and concerns film studio cameras which produce real images and can 
comprise any arbitrary type of camera. The camera may also be stationary or portable and can be 
operated manually or remote controlled (a robot). 

3 0 Fig. 1 shows a film studio where an object 1 0, in this case a car, is filmed by means of a camera 

1 1 (a TV or video camera). The system additionally includes at least one computer unit 12 for 
generating images 13 that can constitute the background or set, a so-called virtual studio. These 
images can also be stored in any memory and fetched when they are needed. Furthermore, the 
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system includes a second computer unit 14 to provide a montage 15, i.e. put together the pictures 
16 and 17 obtained from the cameras 1 1 and the computer unit 12, respectively. Obviously, the 
computer units 1 2 and 1 4 can be integrated into one computer unit. 

It is also possible to transfer the information from several TV -cameras, which generate "real" 
images and bring together them in the computer 14. — 

The system also includes several "tracking cameras", IS, and at least one marker unit 19 arranged 
on the TV-camera 11, which is visible from at least some of the cameras, preferably from at least 
three cameras 18. Preferably, the tracking cameras 18 consist of IR-based CCD-cameras. These 
cameras can be equipped with IR (Infra Read) generating units or communicate with, for example 
IR-flashes. The marker unit consists of one or more markers provided with a reflective surface. 
Moreover, active markers (coded or non-coded) which generate IR (or other type of light) can be 
used. 

For detecting all degrees of freedom of the TV-cameras, three (3) or more CCD-cameras 1 8 are 
used. Preferably, the CCD-cameras operate in 50 and 60 Hz synchronized with synchronization of 
the studio. This is however not a limitation. The function of the CCD-camera is assumed to be 
known for a person skilled in the art and not described in detail herein. 

In an embodiment, the system can include only one tracking camera and one marker arranged on 
the TV-camera, whereby the positioning is carried out according to the disclosed method in the 
Swedish patent No. 9700066-5. 

Fig. 2 shows a schematic marker unit 19 consisting of six spherical or semispherical markers 20 
provided on the ends of carrying rods 21 constituting a rack. The markers are arranged in different 
levels to enable a more accurate determination of it's and consequently the cameras position and 
orientation. It is also possible to use a marker unit including one or more markers. Also, the size 
of the markers in the marker unit can vary. Clearly, the invention is not limited to this 
embodiment of the marker unit and different embodiments may occur. 

In one embodiment, the positioning is carried out through recording IR-reflections (or light) from 
the markers by each CCD-camera. The camera includes an optical element, which projects the 
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image of the marker onto a CCD (Charge Coupled Device) plate in the camera. The surface of 
CCD unit is then scanned and the image signal including pixel information is converted to a 
suitable video signal by means of a converter unit, which can be integrated in the CCD unit. The 
video signal representing the image lines is then sent serially or in parallel to a processing unit. 
The processing unit digitizes the received video signal, for example using an A/D-converter. This 
signal can also be digitized in the camera. — 

The picture elements are arranged in lines by means of a lowpass filter to supply a partly 
continues signal, which can be processed as an analogous signal. However, in the preferred 
embodiment each picture element is measured individually and from the measured values, a value 
is interpolated, which decides when a threshold value is passed. 

The digitalized signal is transferred to a comparator unit, which interpolates the individually 
sampled values about the predetermined threshold value, also called the video level, which is 
obtained from a memory unit. As described above, the aim is to detect when the amplitude of the 
signal passes the threshold value. Each passage represents a start and stop coordinate of each 
segment with a high resolution, which can be about 30 x number of picture elements in a row. In a 
computation unit the center point is computed. 

For a circular marker the area of the image reproduction is computed. Results can later be 
transformed to an interface unit for further transmission to the computer unit, in which the 
computed values (a center point, i.e. x, y, z positions measured with two cameras, area, diameter) 
can be used with respect to disclosed position of the markers. 

By continuously scanning the markers, their motion can be discovered. Camera rotations and 
movements are detected through the different levels of the markers and distance to each other and 
the camera, CCD-cameras 1 8 is arranged to measure the marker units and compute, e.g. six 
degrees of freedom for each marker unit (each marker's x, y, z coordinates, par, tilt and role). 
Before the filming, the image recording unit of the TV-camera, e.g. CCD, is measured in and the 
six generated degrees of freedom are associated to this. Each camera can include a computer unit 
for calculation of the positions of the markers; alternatively, a central unit or the computer unit 14 
is used to perform the image analysis and/or computation. The signal from the CCD-cameras 
(processed or raw) are then sent to the computer 14 to be used for montages. 
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The marker unit 1 9 is preferably arranged on an upper side of the TV-camera and the CCD camera 
hang from a position or roof directed towards the camera. 

Preferably, the calibrated CCD-cameras are situated in a coordinate system specific for the volume 
(studio). Thereby the computed degrees of freedom for the picture element of the TV-camera are 
given in a global coordinate system specific for the volume. These degrees of freedom are - 
transmitted to the computer unit 14 for positioning the camera. 

Through the calibration, the measurement of CCD-cameras can be carried out according to an 
automatic procedure whereby a special marker unit is fastened on the lens mount of the TV- 
camera, after which the CCD-camera system (or the tracking camera system) measures with 
respect to the marker unit relative the marker on TV-camera during the entire measurement 
procedure. 

In a second embodiment shown in fig. 3, a motion sensor 33 is arranged in form of an inertial 
detector or accelerometer and/or gyroscope connected to the camera 31 to be positioned in the 
system. The data from the motion sensor, which can be film data from the camera 3 1 and from the 
positioning cameras 38 via the computer 32 is fed to a computer 34 where x, y, z, par, tilt and role 
are computed (see above), which provides the position of the camera 31. 

Inertial tracking functions both with accelerometers (e.g. for x, y) and can use gyroscope 
(specially for role, par and tilt). It is also possible to use more than one set of x, y, z 
accelerometers provided in different positions on a body to be measured to compute an angel data 
(role, par and tilt). Based on the measurement principle, the accelerometers tend to offset, as the 
position is computed through double integration over the period for the measured acceleration. 
Therefore, small errors are accumulated. However, a high measurement speed, good SNR for the 
signal and good sensitivity also for very small movements of the measured object are obtained. 

The optical system provides a slow and offset free position, while the output of the inertial 
detector has low noise and high measuring speed, preferably higher than the optical system. 
Moreover, the spatial solution for the inertial providing system is higher than the optical. The 
optical system provides absolute measurement (with respect to the spatial coordinates) while the 
inertial-based system provides a relative measurement (offset) and decides whether the object is 
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mobile or stationary. Additionally, it is possible that upon interferences in a system to be able to 
use data from the other system. 

While we have illustrated and described only preferred embodiments of the invention, it is 
5 obvious that several variations and modifications within the scope of the attached claims can 
occur. ~" 



BNSDOC1D <WO O0161?iA1 I > 



WO 00/16121 



PCT/SE99/01589 



DESIGNATION SIGNS 



10 


object 


11,31 


camera 


12, 32 


computer unit 


13 


graphics 


14, 34 


computer unit 


15 


montage 


1 o 


image 


17 


image 


18,38 


tracking camera 


19,39 


marker unit 


20 


marker 


21 


rod 


33 


motion sensor 



WO 00/16121 



10 



PCT/SE99/01589 



CLAIMS 

1. A system in a virtual studio for optical non-contact positioning of at least one camera (11,31) 
in a volume by means of one or more tracking cameras (18, 18), 

characterised in, 

that the camera ( 1 1 , 3 1 ) is equipped with a marker unit ( 1 9, 39), which is visible at least from one 
tracking camera (18, 38), which tracking camera generates a signal containing information 
about a position and/or orientation of the camera (11,31), said information being supplied to a 
unit for further processing with respect to a second set of information for generation of an 
essentially coherent main information set. 

2. System according to claim 1 , 
characterised in, 

that the tracking camera is a CCD equipped camera. 

3. System according to claim 2, 
characterised in, 

that the tracking camera operates within IR range. 

4. System according to claim 1, 
characterised in, 

that the tracking camera communicates with a computer unit (14), which is further connected to 
the camera (1 1 ) for positioning the camera (1 1 ). 

5 . System according to claim 1 , 
characterised in, 

that the tracking camera (18) is provided for measuring the marker unit (19) whereby information 
about degrees of freedom of the marker unit is determined. 

6. System according to claim 1, 
characterised in, 

that the position of the camera is determined relative to a global coordinate system associated with 
the volume. 
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7. System according to any of the preceding claims, 
characterised in, 

that said camera (31) is equipped with at least one motion sensor (33) in form of an inertial 
detector and/or accelerometer and/or gyroscope for an inertial-based positioning. 

8. System according to claim 7, ~ 
characterised in, 

that more then one sets of motion sensors (33) are provided in different positions on a body to be 
measured to compute an angel data (role, pan and tilt). 

9. System according to any of the claims 7 or 8, 
characterised in, 

that the optical positioning provides an absolute measurement while the inertial based positioning 
provides a relative measurement. 

10. System according to any of the claims 1-9, 
characterised in, 

that upon interferences in one of optical or inertial-based positionings, data from the other 
positioning is used. 

1 1 . A method for non-contact positioning of at least one camera (1 1 ) in a volume in a virtual 
studio application by means of one or more tracking cameras (18), 

characterised in 

that the method includes the steps of providing the camera (1 1) with a marker unit (19) visible at 
least from some of the tracking cameras (18), and generating a signal containing information 
about degrees of freedom of the camera (1 1) by means of the tracking camera. 

12. Method according to claim 1 1, 
characterised by 

relative measuring an image recording unit of the cameras (1 1) and generating degrees of freedom 
relative to said relative measurement. 



13. Method according to claim 11, 
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characterised in 

that the measurement of the tracking camera is carried out according to an automatic procedure 
whereby a special marker unit is attached on a lens mount of the camera, after which the CCD 
camera system in the tracking camera system measures with respect to the marker unit relative the 
5 on marker provided on the camera during the entire measurement procedure. 

14. Method according to any of demands 11-13, 
characterised in 

that said camera (31) is provided with at least one motion sensor (33) in form of an inertial 
1 o detector and/or accelerometer and/or gyroscope for an inertial -based positioning. 
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